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Abstract 

To learn from a text, students must make meaningful connections among related ideas in that 
text. This study examined the effectiveness of two methods of improving connections— 
elaborative interrogation and diagrams—in written lessons about posterior probability. 
Undergraduate students (N = 198) read a lesson in one of three questioning conditions (read 
twice, embedded questioning, and elaborative interrogation) and one of three diagram conditions 
(text only, diagram without redundant text, and diagram with redundant text). Elaborative 
interrogation negatively affected learning from the lesson, relative to reading the lesson twice. 
One possible explanation for this finding is that the quality of answers to the elaborative 
interrogations was poor. When the lesson was read twice, diagrams helped learning from the 
lesson relative to text only. Implications of these findings for instruction in probabilistic 
reasoning are discussed. 

Keywords: Diagram, elaborative interrogation, multimedia principle, redundancy principle 
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Learning about posterior probability from written lessons: 

Do diagrams and elaborative interrogation help? 

Probabilistic reasoning is challenging for undergraduate students. However, it is 
important that students leam about probabilistic reasoning so they can be more infonned citizens 
in modern society (Garfield & Ben-Zvi, 2008). One type of probabilistic reasoning task that 
students find particularly difficult is calculating posterior probability (i.e., updating prior 
probability using Bayes’ theorem, such as when detennining a test’s positive predictive value; 
Kahneman & Tversky, 1972; Stanovich & West, 1998; Zieffler, Garfield, Alt, Dupuis, Holleque, 
& Chang, 2008). This task involves considering both the probability of a particular characteristic 
or disease in a given population (i.e., the base rate) and the probability that the identification of 
that characteristic or disease is accurate (i.e., the test accuracy). 

The integration of base rate and test accuracy information can be challenging, 
presumably because of the complexities in connecting the two probabilities correctly (Bar-Hillel, 
1980; Garcia-Retamero & Hoffrage, 2013; Konheim-Kalkstein, 2008). Therefore, techniques to 
improve connections between the two probabilities could help students learn about posterior 
probability. Moreover, students leam more when meaningful connections are made throughout a 
lesson (Kintsch, 1998; McNamara, Levinstein, & Boonthum, 2004). The purpose of this study is 
to examine the effects of two techniques that may promote students making connections among 
ideas within a written lesson: elaborative interrogation and diagrams. 

Elaborative Interrogation 

One technique that may foster connections is elaborative interrogation. Elaborative 
interrogation typically involves “how” or “why” questions intended to prompt students to 
integrate an idea from a written lesson with other ideas from the lesson or with background 
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knowledge (e.g., McDaniel & Donnelly, 1996; Pressley, Symons, McDaniel, Snyder, & Tumure, 
1988; see Dunlosky, Rawson, Marsh, Nathan, & Willingham, 2013, for discussion). Elaborative 
interrogation has been found to promote learning from texts, likely because of the greater 
number of connections students make within the text or between the text and their background 
knowledge (Rouet & Vidal-Abarca, 2002). Thus, it seems likely that elaborative interrogation 
would improve learning from reading a lesson about posterior probability. Indeed, other 
techniques that are intended to foster connection making in ways similar to elaborative 
interrogation, such as self-explanation, have been shown to benefit learning. For example, self¬ 
explanation prompts designed to encourage connections have been found to promote learning 
from probability lessons in which problems are explicitly solved step by step (i.e., worked 
examples; Atkinson, Renkl, & Merrill, 2003). 

It is important to note the similarities and differences between elaborative interrogation 
and self-explanation in order to understand the findings related to these techniques. Elaborative 
interrogation is a specific form of questioning designed to prompt connections within the text or 
between the text and the student’s background knowledge (Pressley et ah, 1998; Ozgungor & 
Guthrie, 2004). In contrast, the term “self-explanation” encompasses a broad range of prompts 
that vary across the literature. In some studies, the prompts are instructions to explain the 
material to oneself (Chiu & Chi, 2014) or explain how a proposed answer is correct or incorrect 
(Rittle-Johnson, 2006). In other studies, the prompts are open-ended questions for the student to 
answer (Leppink, Broers, Imbos, van der Vleuten. & Berger, 2012). Another type of self¬ 
explanation prompt is designed to prompt connections in a manner similar to elaborative 
interrogations (Berthold & Renkl, 2009). Because the term “self-explanation” has been used in 
such varied ways, we use the term “elaborative interrogation” to describe the technique we used 



ELABORATIVE INTERROGATION AND DIAGRAMS 


6 


in the current study because the tenn is more precise than “self-explanation” (see Dunlosky et 
ah, 2013, for a discussion). 

In one form of elaborative interrogation, students are asked to explain why a fact 
presented in the text is true (e.g., Martin & Pressley, 1991; Pressley, McDaniel, Turnure, Wood 
& Ahmad, 1987; Smith, Holliday, & Austin 2010; Woloshyn, Pressley, & Schneider, 1992). This 
type of elaborative interrogation is intended to prompt students to connect facts with their 
background knowledge (Willoughby, Waller, Wood, & MacKinn, 1993). Hence, students with 
higher levels of background knowledge typically accrue greater benefits from this approach than 
do their peers with lower levels of background knowledge (e.g., Pressley et ah, 1987; 

Willoughby et al., 1993). For students with lower levels of background knowledge, a different 
form of elaborative interrogation may be more effective, namely, questions that can be answered 
using information in the text (Ozgungor & Guthrie, 2004). In support of this idea, one study 
showed that students with low levels of background knowledge benefited from self-explanation 
prompts in a statistics lesson only when the answers to the prompts were provided in the lesson 
(Leppink et al., 2012). These findings imply that, for a topic such as calculating posterior 
probability, for which undergraduate students typically lack background knowledge (Beyth- 
Marom, Fidler, & Gumming, 2008; Keeler & Steinhorst, 2001), elaborative interrogation that 
guides students to connect infonnation within the text may be more appropriate than elaborative 
interrogation that guides students to connect information to background knowledge. 

Diagrams 

Another technique that may promote students making connections within a lesson is the 
use of diagrams. According to the multimedia principle, learning is greater when texts are 
accompanied by relevant visuals (Mayer, 2009), because students develop a verbal mental model 
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based on the text and a visual mental model based on the visuals. Students are more likely to 
make connections among different ideas when the information is represented in two distinct 
mental models (Mayer & Gallini, 1990; Mayer, 1999). These connections between the two 
mental models prompt deeper processing of the text and promote learning (Mayer, 2002). 

Diagrams may be particularly useful in lessons about probability. In one previous study, 
performance on probability problems was improved through instruction on constructing 
diagrams (Beitzel & Staley, 2015). In addition, people solve posterior probability problems more 
accurately when diagrams are included, suggesting that diagrams may help people comprehend 
the problems (Garcia-Retamero & Hoffrage, 2013). One study in particular has suggested that 
students benefit from frequency tree diagrams in which probabilities are displayed in frequencies 
when learning how to solve posterior probability problems (Sedhneier & Gigerenzer, 2001). In 
this study, students worked through examples of calculating posterior probability using a 
computer tutorial. Students who received this instruction with diagrams showed remarkable 
gains in accurately calculating posterior probability. In contrast, students who received 
instruction without diagrams showed little improvement. One reason for this could be the 
connections afforded by the presence of verbal and visual representations leading to better 
learning (Mayer, 2009). 

Sedhneier and Gigerenzer’s (2001) instructional method with diagrams was effective, but 
it involved individualized instruction with feedback (e.g., computer tutorials or one-on-one 
tutoring). At the postsecondary level, instructors expect students to learn from reading 
independently (Kim & Anderson, 2011). Reading independently does not provide the 
opportunity for practice and feedback on performance that a computer tutorial can. Therefore, 
there is a need to develop effective lessons from which students can leam when reading 
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independently. Given that making meaningful connections can improve learning from written 
lessons (Kintsch, 1998; McNamara et al., 2004) and that diagrams are most effective when they 
can be connected to relevant text (Atkinson, Derry, Renkl, & Wortham, 2000; Tarmizi & 

Sweller, 1988), techniques for improving connections between diagrams and texts while students 
are reading independently should be examined. 

Making connections between the text and diagram could be easier if there are similar 
surface features in both the text and the diagram (Renkl, 2005). In the case of frequency tree 
diagrams, this would involve having the frequencies in the diagram also provided in the text. 
Indeed, previous work has shown that adding text to diagrams can help guide connections 
between the verbal information and visual information, thereby enhancing learning (Mayer & 
Johnson, 2008; Johnson & Mayer, 2012; Mason, Pluchino, & Tornatora, 2013). On the other 
hand, according to the redundancy principle, information presented in the diagram should not 
also be presented in the text (Sweller, Ayres, & Kalyuga, 2011). This is because presenting 
information in both text and diagram increases the amount of infonnation the student has to 
process, which may overload working memory and diminish learning (e.g., Chandler & Sweller, 
1991). Furthennore, other findings have indicated that, although diagrams with redundant text 
may benefit memory for the lesson, there does not seem to be a benefit for accurate application 
of the ideas in the lesson (McCrudden, Hushman, & Marley, 2014; Ortegren, Serra, & England, 
2015). 

It is possible that the redundant infonnation would actually be helpful for lessons about 
probabilistic reasoning with diagrams. Previous studies in which the redundant information led 
to less learning used complex diagrams for scientific concepts with detailed information that may 
be difficult to communicate through text (e.g., Holliday, 1976). In contrast, it is possible that 



ELABORATIVE INTERROGATION AND DIAGRAMS 


9 


having that having the simple, numeric information in the text that is redundant with the diagram 
would serve a function similar to labels and could improve connections between the text and 
diagram. In this way, it is possible that text redundant with the diagram could increase learning 
from the lesson. 

Elaborative Interrogation with Diagrams 

It stands to reason that diagrams might be particularly beneficial when elaborative 
interrogation is included in the lesson. Students often attend to visual representations less than 
they attend to text (Schuler, Scheiter, Rummer, & Gerjets, 2012; Schwonke, Berthold, & Renkl, 
2009), and without sufficient attention to visual representations, students cannot connect visual 
and verbal information. Because connections between visual and verbal infonnation are critical 
for meaningful learning (Cromley, Snyder-Hogan, & Luciw-Dubas, 2010; Mayer, 2009), 
methods that prompt students to integrate verbal and visual infonnation in text may promote 
learning. Elaborative interrogation may be particularly useful for guiding students to attend to the 
visual representations, and then to integrate visual and verbal infonnation (Aleven & Koedinger, 
2002). This could be accomplished by posing elaborative interrogations that encourage students 
to connect information in the diagram and the text, in addition to elaborative intenogations that 
prompt connections within the text. This idea is supported by previous work indicating that self¬ 
explanation prompts can foster students making connections between relevant information in text 
and diagrams in worked examples (Berthold & Renkl, 2009; Berthold, Eysink, & Renkl, 2009). 

If elaborative interrogation is beneficial for learning from a lesson on posterior probability, this 
benefit might be enhanced when elaborative interrogation is coupled with diagrams. 


The Current Study 
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The purpose of this study was to examine the effects of elaborative interrogation and 
diagrams on learning about posterior probability from a lesson that they read independently. 
Learning was assessed with posterior probability problems on which the logic presented in the 
lesson could be applied. We predicted that students would learn more with elaborative 
interrogation. As in past work (e.g., Ozgungor & Guthrie, 2004; Pease, 2012; Smith et ah, 2010), 
we included two additional conditions to allow us to better assess the effects of elaborative 
interrogation. To control for time on task (i.e., time with the lesson), one group of students read 
the lesson twice. To control for the effects of simply answering questions (e.g., McMaster et ah, 
2012), a second group of students answered simple “who” or “what” questions about specific 
ideas explicitly stated in the lesson. These questions could be accurately answered with only 
shallow processing of the lesson material. By comparing perfonnance in the elaborative 
interrogation condition against performance in these two other conditions, we could be confident 
that any observed effects of elaborative interrogation were due to these questions prompting 
deeper processing, rather than time with the lesson or the simple act of answering questions. 

In line with previous research on visual representations (e.g., Beitzel & Staley, 2015; 
Mayer, 2009), we also predicted that students would leam more if their assigned lesson included 
a diagram. However, we had competing predictions as to whether learning would be greatest if 
the infonnation presented in the diagram was also presented redundantly in the text, or if that 
information was not presented redundantly in the text. Based on previous work with adding text 
to diagrams (Johnson & Mayer, 2012; Mason et al., 2013), including text redundant with the 
diagram may enhance learning. In contrast, based on the redundancy principle (Sweller et al., 
2011), including text redundant with the diagram could decrease learning. However, we expected 
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that diagrams would enhance learning relative to text alone, regardless of the presence of 
redundant text. 

If we find that elaborative interrogation and diagrams are beneficial for student learning, 
we might find that combining elaborative interrogation with diagrams would enhance their 
effects. Some of the elaborative interrogations in the lessons with diagrams were designed to 
prompt connections between visual information in the diagram and verbal information in the 
text. Connecting information in visual and verbal representations has previously been found to be 
beneficial for learning (Cromley et ah, 2010; Mayer, Steinhoff, Bower, & Mars, 1995; Mayer, 
2009). Hence, we predicted that the lesson that combined diagrams and elaborative 
interrogations would lead to optimal learning. 

We were also interested in how difficult the lessons were for students. Difficulty was 
assessed through ratings as well as through time spent on the lesson. Because answering 
elaborative interrogations likely requires more effort than reading twice or answering simple 
questions, we anticipated that lessons with elaborative interrogations would be more difficult. 
However, we expect that elaborative interrogation would be a desirable difficulty leading to 
increased learning (McDaniel & Butler, 2011; Richland, Bjork, Finley, & Linn, 2005). 
Knowledge of how difficult the lessons were for students could be useful in interpreting our 
findings for redundant text. Recall that the redundancy principle states that infonnation should be 
presented in the diagram or the text, because including information in both the diagram and the 
text would be extra, unnecessary information for students to process (Sweller et ah, 2011). 

Hence, this principle would be supported if students report that lessons with redundant text were 
more difficult than lessons without redundant text. In contrast, previous work has shown that 
adding redundant text to diagrams through labeling may make it easier to connect the text and 
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diagram (Johnson & Mayer, 2012). Therefore, students may find that diagrams with redundant 
text are less difficult than diagrams without redundant text. Overall, the difficulty ratings in 
conjunction with performance on the learning measures could inform our understanding of the 
effects of elaborative interrogations and diagrams. 

Methods 

Participants 

Participants were 248 undergraduates who earned extra credit in their introductory 
psychology course for participation. Three participants left the study before completing the 
posttest; their data were excluded. Of the remaining 245 participants, there were 158 females and 
86 males (1 participant did not report gender) with an average reported age of 19.05 years (SD = 
2.89 years). English was reported as the native language of 164 participants; 80 participants 
reported a native language other than English (1 participant did not report a native language). 
Regarding ethnicity, 156 were Caucasian, 64 were Asian, 9 were Latino, 5 were African- 
American, 1 was Native American, and 5 were biracial or multiracial (5 participants did not 
report ethnicity). 

Materials 

The lesson was adapted from a textbook by Heuer (1999) and incorporated an example 
from Sedlmeier and Gigerenzer’s (2001) experiments. There were two examples of scenarios in 
which base rate and specificity probabilities were presented, and the posterior probability was 
calculated using natural frequencies. For the diagram conditions, there was a frequency tree 
diagram for each of the two examples. For the diagram-without-redundant text condition, some 
of the infonnation in the diagram was not also presented in the text. Specifically, the frequency 
information provided in the diagram was not in the text. For the diagram-with-redundant-text 
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condition, the frequency information in the diagram was also presented in the text (see Appendix 
for lesson excerpts from each diagram condition). The texts for the text-only and diagram-with- 
redundant-text conditions were identical, with the exception that the diagram-with-redundant text 
condition contained a sentence directing the readers’ attention to the diagram. The text-only 
condition was 987 words long, with a Flesch-Kincaid Grade Level of 12.0 and Flesch Reading 
Ease of 40.9. The text for the diagram-without-redundant-text condition was 979 words long, 
with a Flesch-Kincaid Grade Level of 12.0 and Flesch Reading Ease of 41.7. The diagram-with- 
redundant-text condition was 1003 words long with a Flesch-Kincaid Grade Level of 12.0 and 
Flesch Reading Ease of 41.3. 

For the two question conditions, (i.e., the elaborative-interrogation and embedded- 
questioning conditions), there was one question after approximately every paragraph, for a total 
of six questions. For the diagram and questioning conditions, two of the six questions were asked 
after the diagrams in the lesson. For the diagram-without-redundant-text condition, answering 
the two questions after diagrams correctly required using information from the diagram. 
However, for the diagram-with-redundant-text condition, the questions could be answered using 
information from either the text or the diagram (or both). 

Elaborative interrogation questions were complex “how” and “why” questions (e.g., 

“Why does including the base rate make the probability a patient with fever, chills, and skin 
lesions has sepsis 47%?” and “Why is the base rate fallacy such a common mistake?”). 
Following Ozgungor and Guthrie (2004), the answers to elaborative interrogations were implicit 
in the text; the participant had to connect different ideas presented in the text and/or diagrams. 
Therefore, participants did not need to use background knowledge to answer the elaborative 
interrogation prompts. In contrast, embedded questions were simple (e.g., “What is the base 
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rate?” and “How many patients with sepsis do not have high fever, chills, and skin lesions?”), 
and their answers were explicitly stated in the text and/or clearly presented in the diagrams. At 
the beginning of each lesson was a text reminder that the participant would be asked to answer 
questions based on the information in the lesson after reading the lesson. The lesson in the read- 
twice condition also began with the text reminder that participants needed to read the lesson 
twice. At the end of the read-twice condition lesson, participants were reminded that if they had 
only read the lesson once, they needed to re-read the lesson starting at its beginning. 

Measures 

Pretest. Prior to reading the lesson, participants were asked to solve the following 
posterior probability problem adapted from Bramwell, West, and Salmon (2006): “All medical 
tests have error. For example, the serum test screens pregnant women for fetuses with Down 
syndrome. The test is a very good one, but not perfect. Roughly 100 fetuses out of 10,000 have 
Down syndrome. Of these 100 fetuses with Down syndrome, 90 pregnant women will have a 
positive test result. Of the remaining 9,900 unaffected fetuses, 99 pregnant women will still have 
a positive test result. What is the probability a pregnant woman who has a positive result on the 
test actually has a fetus with Down syndrome?” The problem was presented only as text; no 
visuals were included. 

Difficulty self-report. Following the lesson, participants were asked to rate their level of 
agreement with the statement “The lesson I just read was difficult” on a Likert scale from 1-7. 

Comprehension assessment. We developed a comprehension assessment based on 
previously used sentence verification (Royer, 2001) and inference verification (Wiley & Voss, 
1999) techniques. The assessment consisted of 13 sentences. Seven of the sentences were correct 
or incorrect paraphrases (i.e., contained or contradicted information explicitly stated in the 
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lesson) and six of the sentences were correct or incorrect inferences (i.e., based on information in 
lesson that was not explicitly stated). These paraphrases and inferences were based on ideas that 
were evenly distributed from throughout the lesson. Participants were asked to indicate whether 
each sentence was consistent or inconsistent with what they just read. Internal consistency for the 
comprehension assessment was unacceptable for the paraphrases (Cronbach’s a = .34), 
unacceptable for the inferences (Cronbach’s a = .30), and poor for the overall measure 
(Cronbach’s a = .50). Because of the low internal consistency, the comprehension assessment 
was not used as a learning measure and it is not discussed further. 

Posttest. A posttest was used to assess learning of calculating posterior probability. The 
posttest consisted of 4 posterior probability problems similar to the one on the pretest. The 
problems were presented in text only; no visuals were included. Internal consistency for the 
posttest was good (Cronbach's a = .87). 

Procedure 

Participants were randomly assigned to one of nine conditions, reflecting a 3 (diagram 
condition: diagram with redundant text, diagram without redundant text, text only) x 3 
(questioning condition: elaborative interrogation, embedded questions, read-twice) factorial 
design. One to six participants came to the testing room for a given experimental session, and 
they engaged in the experimental tasks individually. All participants in each small group were in 
the same condition. All tasks were paper-and-pencil (i.e., nothing was administered on a 
computer). After providing informed consent, participants were given one posterior probability 
problem to solve as a pretest. Then, if they were in one of the questioning conditions (i.e., 
elaborative interrogation or embedding questioning), they were given the lesson and an answer 
sheet for their questions. The experimenter instructed participants to read the lesson carefully 
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because they would be asked to answer questions and solve problems based on its infonnation. 
Participants in the read-twice condition were instructed to read the lesson twice. The 
experimenter monitored the participants to ensure that they followed directions and read the 
lesson twice. Participants in the questioning conditions were instructed to answer the numbered 
questions in the lessons in the appropriate places on the answer sheet. Participants were 
pennitted to look back to previously-read text, but they were not specifically instructed to do so. 
When participants finished reading the lesson, they returned it to the experimenter and completed 
the self-report of lesson difficulty. Then, they solved 21 simple multiplication and division 
problems as a distracter task, and then completed comprehension assessment followed by the 
posttest. Finally, participants self-reported their demographic infonnation. All of the tasks were 
completed in a single session, which was approximately 45-60 minutes in length. The 
experimenter recorded the time participants received the lesson as well as the time they returned 
the lesson to the experimenter. 

Scoring 

Pre and posttests. Pre and posttests were scored for accuracy. An accurate answer was 
the number of true positives over the combined number of true positives and false positives for 
any given problem. Pretest scores ranged from 0-1, as the pretest had only one item. The 
maximum posttest score was 4. 

Difficulty. The participant’s response on the Likert scale was used as a measure of 
perceived difficulty. 

Answers to elaborative interrogation questions. Answers to elaborative interrogation 
questions were scored using a rubric that contained the following categories: ideal, adequate, 
inadequate, circular, wrong, and missing. Elaborative interrogation questions were designed to 
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prompt integration of concepts within the lesson. Therefore, an answer was considered ideal if it 
correctly included three or more concepts presented in the lesson. Adequate answers correctly 
included two concepts. Inadequate answers correctly included only one concept. Circular 
answers repeated the information in the question (e.g., El: “Why is the base rate important when 
calculating probability?” Circular Answer: “Because it is important infonnation you need to use 
to calculate probability.”). Wrong answers provided incorrect or inappropriate infonnation. 
Missing answers included responses such as “I have no idea” in addition to bla nk responses. Two 
research assistants coded answers to elaborative interrogations. Twenty-five percent of the 
answers were coded in common with excellent reliability (Cohen’s k = .84). Disagreements were 
resolved through discussion. 

To convert the rubric to scores, ideal answers were scored as ‘3’, adequate answers were 
scored as ‘2’, and inadequate answers were scored as ‘1.’ Missing, incorrect, and circular 
answers were scored as ‘0.’ The scores for all 6 of the elaborative interrogations were summed 
for a measure of answer quality. The maximum possible score was 18. 

Answers to embedded questions. Because embedded questions were much simpler than 
elaborative interrogations, their answers were scored dichotomously as correct or incorrect. As 
with the elaborative interrogations, two research assistants coded answers to embedded 
questions. Twenty-five percent of the answers were coded in common with excellent reliability 
(Cohen’s k = .98). Disagreements were resolved through discussion. Correct answers were 
scored as ‘ 1’ and incorrect answers as ‘0.’ The number of correct answers was the total score. 

The maximum possible score was 6. 


Results 
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Table 1 presents descriptive statistics for pretest scores by condition. Prior to testing the 
hypotheses, we examined the a priori distribution of pretest scores by condition. Although the 
distribution of pretest scores did not differ significantly as a function of diagram condition, % 2 (2, 
N = 243) = 5.04, p = .08, or questioning condition, % 2 (2, N = 243) = .62,p = .73, the difference 
in pretest scores by condition could still confound results. For this reason, participants who 
answered the pretest question correctly (N = 47) were removed from analyses. To test the effects 
of diagram and questioning on learning from the lesson, we used logit mixed models, as 
recommended by Jaeger (2008; see Snijders & Bosker, 2012 for more infonnation about 
multilevel models, including logit mixed models), implemented using the package lme4 in the R 
statistical software (Bates, 2010; Bates, Maechler, & Bolker, 2012). Pair-wise corrections were 
not necessary because of the use of multilevel modeling through hne4 (Gelman, Jill, & Yajima, 
2012). For all analyses, we set the Type I error rate at a = .05. We report Type III Wald chi- 
square tests of the parameter estimates against 0. 

Did questioning and/or diagrams affect posttest accuracy? 

We first examined the effects of the questioning and diagram conditions on posttest 
accuracy. Based on previous research findings (Ozgungor & Guthrie, 2004; Pressley et al., 1987; 
Smith et al., 2010), we expected that elaborative interrogation would improve learning. Based on 
the multimedia principle, we expected that diagrams would improve application of the lesson 
content, as indicated by accuracy on posttest questions, relative to lessons with only text. We had 
competing predictions regarding the effects of redundant text with the diagram. Based on the 
redundancy principle, it is possible that redundant text would add unnecessary information to 
process thereby decreasing learning. However, it is also possible, based on previous work with 
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labeling diagrams (Johnson & Mayer, 2012) that text redundant with the diagram would facilitate 
connections between the text and the diagram, thereby increasing learning. 

To address these hypotheses, we used a mixed-effects model with questioning and 
diagram condition as fixed factors, problem and participant as random factors, and accuracy on 
each item as the dependent variable. Figure 1 presents the mean posttest item accuracy by 
condition. In our analysis, we first considered the effects of diagram presence , by collapsing the 
two conditions that involved diagrams, and comparing them to the text only condition. We then 
compared the two conditions that included diagrams (diagram with redundant text and diagram 
without redundant text) to one another. 

Contrary to hypotheses, there was no main effect of diagram presence on posttest 
accuracy, Wald y^{2, N = 198) =1.43,/? = .15. However, there was a significant interaction of 
diagram presence and questioning condition, Wald x 2 (4, N = 198) = 6.89 ,p = .03. In the read- 
twice condition, participants whose lessons included diagrams performed better on the posttest 
than participants whose lessons did not include diagrams, 5 = 3.11, Wald Z= 2.29, p = .02. 
However, diagram presence did not affect posttest performance for participants in the embedded 
questioning condition, B = .19, Wald Z= .12, p = .91, or the elaborative interrogation condition, 

B = .86,Wald Z = .28, p = .60. 

Questioning condition had a main effect on posttest item accuracy, Wald % 2 (2, N = 198) 

= 13.73, p < .001. Unexpectedly, however, participants in the elaborative interrogation condition 
had lower posttest item accuracy than did participants in the read-twice condition, B = 3.75, 

Wald Z = 13.81,/? < .001. There were no differences between the elaborative interrogation and 
embedded questioning conditions, B = .81, Wald Z= .50,/? = .62. 
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We next examined whether the redundancy of the diagram with the text influenced 
posttest performance. To address this question, we used a mixed-effects model to compare the 
diagram with redundant text and diagram without redundant text conditions (i.e., excluding the 
text-only condition). There was no main effect of diagram condition, Wald %2(2, N = 198) = .78, 
p = .44 and no interaction with questioning condition, Wald y2(A, N = 198) = 1.30,/? = .52. 

Thus, performance was not affected by the redundancy of the text and diagram; participants 
performed similarly with and without redundant text. 

The negative effect of questioning condition was also evident in this analysis, Wald y2(2, 
N = 198) = 16.27,/? < .001. However, in this subsample, the benefit of reading twice compared 
to elaborative interrogation was not reliable, B = 3.33, Wald Z= 1.68,/? = .09. The elaborative 
interrogation and embedded questioning conditions did not differ, B = -.89, Wald Z= 00,/? = .99. 

In brief, contrary to what was predicted, elaborative interrogation was not beneficial for 
learning how to solve posterior probability problems. Indeed, reading twice yielded higher scores 
on the posttest than elaborative interrogation. Diagrams also had limited effects; they helped only 
when participants read the lesson twice, and not when they answered questions. Redundant text 
did not appear to affect posttest accuracy for students whose lessons had diagrams. 

Did diagrams and questioning affect lesson difficulty and time with the lesson? 

To better understand the nature of the posttest findings, we also examined the effects of 
diagrams and elaborative interrogations on lesson difficulty ratings and time with the lesson (i.e., 
the number of minutes spent with the lesson). The number of minutes spent with the lesson was 
positively skewed, so a square-root transfonnation was applied prior to analysis (Osborne, 2002). 
Because there was only one rating of difficulty or measure of time with the lesson, the multilevel 
modeling analyses conducted with posttest item accuracy were not appropriate (Snijders & 
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Bosker, 2012). Instead, we conducted ANOVAs with diagram condition and questioning 
condition as fixed factors and lesson difficulty ratings and minutes spent on the lesson as the 
dependent variables. Bonferonni corrections were applied to pairwise comparisons. 

As with posttest accuracy, we examined the effect of diagram in two ways. The first was 
to examine diagram presence by collapsing the two diagram conditions and comparing them to 
text alone. The second was to compare the two diagram conditions to one another. We expected 
that diagrams would make the lessons easier to comprehend, thereby yielding lower difficulty 
ratings and less time with the lesson. We did not have specific predictions regarding the effects 
of redundancy of text and diagram. In addition, we expected that elaborative interrogation would 
increase the amount of effort exerted on the lesson, thereby yielding higher difficulty ratings and 
more time spent with the lesson. 

Lesson Difficulty. Figure 2 presents the mean difficulty ratings by condition. We first 
examined whether diagram presence and questioning condition affected difficulty ratings. There 
was no main effect of diagram presence on lesson difficulty ratings, F(\ , 196) = 1.16,/? = .28. 
However, there was a significant interaction, F( 2, 192) = 3.04,/ = .05. In the read-twice 
condition, participants whose lesson included a diagram rated the lesson as easier than those 
whose lesson did not include a diagram, F(l, 75) = 5.76,/ = .02, Cohen’s d= .55. In the 
embedded questioning and elaborative interrogation conditions, presence of the diagram did not 
affect difficulty ratings, F( 1, 54) = .65,/ = .43 and F( \ , 62) = 1.24,/ = .27, respectively. 

There was also a main effect of questioning condition, F( 2, 195) = 7.3,/ = .001. 
Participants in the elaborative-interrogation condition rated the lesson as more difficult than did 
participants in the read-twice condition, / = .001, Cohen’s d = .59. In addition, participants in the 
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embedded-questioning condition also rated the lesson as more difficult than did participants in 
the read-twice condition,/? = .01, Cohen’s d = .63. 

Next, we examined whether redundant text affected difficulty ratings for lessons with 
diagrams. As with posttest item accuracy, there was no effect of text redundancy on difficulty 
ratings by students whose lessons had diagrams, F( 1, 119) = .05, p = .82. In addition, there was 
no interaction between text redundancy and questioning, F( 2, 114) = .90,/? = .41. Again, there 
was a main effect of questioning condition, F( 2, 117) = 7.65,/? = .001. Participants in the 
elaborative-interrogation condition rated their lessons as more difficult than did participants in 
the read-twice condition,/? = .001, Cohen’s d = .88. Participants in the embedded questioning 
also rated their lessons as more difficult than participants in the read-twice condition, but this 
difference was not reliable,/? = .06. 

Time with lesson. Figure 3 presents the mean number of minutes spent on the lesson by 
condition (means reported are not square-root transfonned in order to provide the reader with 
meaningful descriptive statistics). There was no effect of diagram presence on the number of 
minutes spent with the lesson, F(l, 196) = 1.40,/? = .24, and no interaction between diagram 
presence and questioning condition, F( 2, 192) = .97, p = .38. However, there was a main effect 
of questioning condition, F( 2, 195) =68.21 ,P< .001. Participants in the elaborative-interrogation 
condition spent more time with the lesson than did participants in either the read-twice condition, 
p < .001, Cohen’s d = 1.48, or the embedded-questioning condition, p = .001, Cohen’s d = .40. In 
addition, participants in the embedded-questioning condition spent more time with the lesson 
than did participants in the read-twice condition, p < .001, Cohen’s d = 1.08. 

When examining the effects of redundant text on the time spent with lessons with 
diagrams, the findings are similar to those for difficulty ratings. Redundant text did not affect 
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time spent with the lesson, F( \, 119) = .09, p = .77. In addition, there was not a reliable 
interaction between redundancy and questioning, F( 2, 114) = 2.95, p = .06. As with previous 
analyses, there was main effect for questioning, F(2, 117) = 41.41 ,P< .001. Participants in the 
elaborative-interrogation condition spent more time with their lessons than either participants in 
the read-twice condition,/; < .001, Cohen’s d= 1.41 or participants in the embedded-questioning 
condition,/? < .001, Cohen’s d = .25. In addition, participants in the embedded-questioning 
condition spent more time with their lessons than did participants in the read-twice condition, p 
= .001, Cohen’s d= 1.16. 

Overall, based on difficulty ratings and time spent with the lessons, diagrams had little 
effect on making the lessons easier to understand. However, the lesson difficulty ratings and 
number of minutes spent on the lesson clearly indicate that answering elaborative interrogations 
while reading was difficult and time consuming for students. In addition, reading twice, although 
intended to control for time with the lesson to a certain degree relative to elaborative 
interrogation, was clearly not an appropriate control for time on task. Had elaborative 
interrogation yielded a benefit relative to reading twice, time with the lesson would have been an 
important variable to include in the analyses of learning measures. 

Answer Quality for Questioning Conditions 

There were two important findings regarding questioning. The first was that reading 
twice was more beneficial for learning how to calculate posterior probability than was 
elaborative interrogation. The second was that elaborative interrogation did not foster learning 
more than embedded questions. To better understand these findings, we examined the quality of 
the answers to the questions in the elaborative-interrogation and embedded-questioning 
conditions (see Scoring section for scoring criteria). 
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Elaborative Interrogations. Participants received an average score of 4.80 (SD = 2.27), 
out of a maximum possible score of 18, for the answers to elaborative interrogations. 
Approximately 37% of the answers were missing, circular, or wrong, 51% of the answers were 
inadequate, 9% of the answers were adequate, and 4% were ideal. We used a Spearman’s rank- 
order correlation to test for associations between accuracy of answers to the elaborative 
interrogations and the sum of accurate answers on the posttest. There was a positive correlation 
between elaborative interrogation answer scores and posttest scores, p(62) = .40,/? = .001. 

Embedded Questions. Participants answered an average of 3.08 embedded questions 
correctly (SD = 1.70) out of 6 questions total. As with the elaborative interrogations, we used a 
Speannan’s rank-order correlation to test for an association between the accuracy of the answers 
to embedded questions and the posttest scores. There were no association between embedded 
question answer accuracy and posttest scores, p(54) = .11,/? = .44. 

Discussion 

The purpose of this study was to examine the effects of elaborative interrogations and 
diagrams on learning from a written lesson on calculating posterior probability. Both elaborative 
interrogation and diagrams were both expected to improve the meaningful connections students 
made in the lesson, thereby increasing learning. 

Effects of elaborative interrogation on learning 

We expected that elaborative interrogation would benefit learning from the lesson, because 
of an anticipated increase in meaningful connections among ideas in the lesson (e.g., McDaniel 
& Donnelly, 1996; Ozgungor & Guthrie, 2004). Contrary to this expectation, answering 
elaborative interrogation questions led to less learning than reading the lesson twice. One 
possible explanation for this finding on is that the quality of the answers students provided was 
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generally poor. According to Jiang and Elen (2011), answering questions while reading improves 
learning from text only if students answer the questions as they were intended. For the 
elaborative interrogations in this study, students generally did not provide answers indicative of 
the connections among ideas in the lesson these questions were intended to stimulate. The 
overwhelming majority of the answers to the elaborative interrogation questions were missing, 
circular, wrong, or inadequate. Given these findings regarding answer quality, it is not surprising 
that elaborative interrogation did not promote learning. These findings are consistent with 
previous work in which incorrect elaborations for self-explanations while learning were 
associated with poor problem-solving perfonnance (Berthold & Renkl, 2009). 

We also expected that elaborative interrogation would add difficulty to the lesson. 
However, we anticipated that this difficulty would be desirable and increase learning (Richland 
et al., 2005). As expected, we found that students reported higher difficulty ratings and spent 
more time with lessons that incorporated elaborative interrogations. Based on the learning 
measures, however, this increased difficulty did not appear to be desirable. Thus, the level of 
difficulty imposed by the elaborative interrogations may be another explanation for our findings. 
The logical reasoning behind posterior probability is quite complex; students may have been 
overwhelmed by the elaborative interrogations. 

Effects of diagrams on learning 

In this study, we sought to improve student learning on posterior probability problems by 
incorporating diagrams into lessons. Based on the multimedia principle, which states that 
learning is improved when relevant visuals are incorporated into lessons (Mayer, 2009), we had 
expected that including diagrams in the lesson would improve learning. We found a benefit of 
diagrams, but only for the read-twice condition. These findings are somewhat consistent with the 
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multimedia principle in that the benefits of diagram were limited to a particular condition. 
Although the benefits of diagrams were evident only when reading twice, this benefit was likely 
due to the connections students generated between the verbal information in the text and the 
visual information in the diagrams. 

We also examined the effect of including text redundant with the diagram. Based on the 
redundancy principle, which states that information should be presented in the text or diagram, 
but not both (Sweller et ah, 2011), one prediction was that omitting text redundant with the 
diagram would enhance learning. Alternatively, add text redundant with the diagram could assist 
students in making connections between the text and diagram in a manner similar to labeling 
(Mason et al., 2013). Hence, a competing prediction was that including text redundant with the 
diagram would improve learning more than omitting text redundant with the diagram. However, 
we did not find a reliable difference for text redundancy for either learning or difficulty. One 
reason could be that the possible benefits of redundant text evened out the possible negative 
effects of redundant text, thereby causing no overall difference. 

Elaborative Interrogation with Diagrams 

The elaborative interrogation questions were designed to promote connections between 
the text and diagrams, thereby increasing learning. However, the elaborative interrogation 
questions appear to have attenuated any benefit the diagram may have provided. Students may 
have been too distracted by generating answers to the elaborative interrogation questions to 
develop meaningful connections between the visual and verbal information. 

Pedagogical Implications 

Given that some of the findings are inconsistent with previous work (e.g., Ozgungor & 
Guthrie, 2004; Smith et al., 2010), it is premature to indicate implications. However, if this 
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study would be supported by future work with similar findings, there would be implications for 
educational practice, both in probability instruction and more generally. At a general level, our 
findings indicate that relevant visual information, such as diagrams, in texts can improve 
learning—though it does not do so under all instructional conditions. In this study, diagrams 
fostered students’ application of the lesson content relative to text alone only if students read the 
lesson twice, and not if they answered questions while reading. Taken together, these findings 
suggest that visual representations may be beneficial, but only if students have adequate 
resources available to process them. Teachers may find it beneficial to incorporate relevant 
visual representations in their lessons, and to encourage students to construct visual 
representations when solving problems. However, teachers may also need to consider the 
resource demands of working with visual representations. 

Our findings suggest that elaborative interrogation does not necessarily prompt 
meaningful connections among ideas in the text that would improve learning of material. 
However, the results indicate that better answers to elaborative interrogations were associated 
with better learning from the lesson. Therefore, teachers may find it useful to model appropriate 
answers and monitor answer quality when using elaborative interrogation with their students. 
Limitations and Future Directions 

Some limitations of this study that should be noted. Namely, we did not address the 
possible influence of background knowledge because the topic in this study, posterior 
probability, is one with which this population typically has little background knowledge 
(Morsanyi & Handley, 2012). For this reason, the answers to the elaborative interrogations were 
specifically designed to not require background knowledge (see Ozgungor & Guthrie, 2004, for 
similar methodology). However, background knowledge may have provided support to students 
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to help them understand the material and effectively answer the elaborative interrogations. In 
other words, students may have had the cognitive capacity to provide suitable answers to the 
elaborative interrogations if the lesson was about a topic in which they had a good amount of 
background knowledge (Renkl, Atkinson, & GroPe, 2004). A future study on a topic in which 
students have stronger background knowledge may find that the combined use of elaborative 
interrogation and diagram enhances learning. In addition, there was only one reliable measure of 
learning in this study that assessed how to solve the problem (i.e., procedural knowledge), but 
not why the problem should be solved a particular way (i.e., conceptual knowledge). Given the 
importance of conceptual knowledge (Crooks & Alibali, 2014), a study in which measures of 
conceptual learning are included would be infonnative. 

Overall, answer quality for elaborative interrogation questions was quite poor in this 
study. We propose two approaches to improve answer quality in pedagogical contexts and in 
future studies. The first is practice training, in which students receive examples of what types of 
answers are expected of them. This practice training would provide a model for the students, 
which could improve their answer quality (see Domisch, Sperling, & Zeruth, 2011, for 
discussion). The second is to provide students with feedback on their answers, either directly, 
through their peers while working collaboratively (after training), or through computer-based 
tutoring. Such feedback has been shown to enhance the effectiveness of answering questions 
while reading a lesson (Andre & Thieman, 1988; Garcia-Rodicio, 2014; Peverly & Wood, 2001). 
Moreover, feedback could provide motivation for students to construct high-quality answers to 
elaborative interrogations (Biggs & Tang, 2011). If these ideas could effectively improve answer 
quality, positive effects of elaborative interrogation might be seen. 


Conclusion 
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Posterior probability, like many probabilistic reasoning concepts, is frequently 
challenging for students to learn. In this study, we sought to improve student learning on 
posterior probability problems by incorporating diagrams and elaborative interrogations in 
lessons that students read independently. Our findings regarding elaborative interrogations 
indicate that the difficulty of the material should be considered when using this technique. We 
believe that poor answer quality is one reason for the negative effects of elaborative interrogation 
that we observed, and we have suggested some approaches that might improve answer quality. 
Our finding that diagrams promoted learning is consistent with the multimedia principle (Mayer, 
2009). In sum, this work provides some guidance about how to improve student learning about 
probabilistic reasoning, while also highlighting the challenges inherent in this complex domain. 
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Appendix 

Lesson Excerpts 

Text Only 

It’s helpful to imagine 100 patients that come to the walk-in clinic. According to the base 
rate, we know that 10 of the patients will have sepsis and 90 will not. According to the specific 
case infonnation, we know that 80% or 8 of the 10 patients with sepsis will have high fever, 
chills, and skin lesions and 20%, or 2 of the 10 patients will not. Also, of the 90 patients who 
don’t have sepsis, 10% or 9, will have high fever, chills, and skin lesions, and 90%, or 81, will 
not have these symptoms. 

This makes a total of 17 patients with high fever, chills, and skin lesions, and 83 patients 
without these symptoms, of which 8 of the 17 actually have sepsis; the other 9 have the 
symptoms without actually having sepsis. Therefore, when a doctor sees a patient with the 
symptoms of sepsis (high fever, chills, and skin lesions), the probability that the patient actually 
has sepsis is only 8/17ths or 47%. 

Diagram without Redundant Text 

It’s helpful to imagine 100 patients that come to the walk-in clinic. According to the base 
rate, we know how many of these patients will have sepsis and how many will not. According to 
the specific case information, we know that 80% of patients with sepsis will have high fever, 
chills, and skin lesions and 20% of the patients with sepsis will not. Also, of the 90 patients who 
don’t have sepsis, 10% will have high fever, chills, and skin lesions, and 90% will not have these 
symptoms. This is shown in the diagram. 
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This makes a total of 17 patients with high fever, chills, and skin lesions, but not all of 
these patients have sepsis. Therefore, when a doctor sees a patient with the symptoms of sepsis 
(high fever, chills, and skin lesions), the probability that the patient actually has sepsis is only 
8/17ths or 47%. 

Diagram with Redundant Text (redundant frequency information in bold) 

It’s helpful to imagine 100 patients that come to the walk-in clinic. According to the base 
rate, we kn ow that 10 of the patients will have sepsis and 90 will not. According to the specific 
case information, we know that 80% or 8 of the 10 patients with sepsis will have high fever, 
chills, and skin lesions and 20%, or 2 of the 10 patients will not. Also, of the 90 patients who 
don’t have sepsis, 10% or 9 , will have high fever, chills, and skin lesions, and 90%, or 81 , will 
not have these symptoms. This is shown in the diagram. 
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This makes a total of 17 patients with high fever, chills, and skin lesions, and 83 patients 
without these symptoms, of which 8 of the 17 actually have sepsis; the other 9 have the 
symptoms without actually having sepsis. Therefore, when a doctor sees a patient with the 
symptoms of sepsis (high fever, chills, and skin lesions), the probability that the patient actually 
has sepsis is only 8/17ths or 47%. 
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a Diagram without 
Redundant Text 
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Read Twice Embedded Elaborative 
Questioning Interrogation 

Figure 1. Posttest item accuracy by condition (means with standard error bars; N = 198). 
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□ Text Only 


□Diagram without 
Redundant Text 

□Diagram with 
Redundant Text 


Read Twice Embedded Elaborative 
Questioning Interrogation 

Figure 2. Difficulty ratings by condition (means with standard error bars; N = 198). 
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Questioning Interrogation 


□ Text Only 

□Diagram without 
Redundant Text 

□Diagram with 
Redundant Text 


Figure 3. Number of minutes with the lesson by condition (means with standard error bars; N = 
198). 




























